Image processing apparatus and image processing method

ABSTRACT

In an image processing apparatus, when a plurality of items of moving-image data, modified data obtained by modifying the plurality of items of moving-image data, and presentation-description data which indicates the presentation order of the plurality of items of moving-image data and the modified data, are processed, an editing screen formed of representative images of moving-image data for which presentation is specified in the presentation-description data, and of a representative image of the modified data is generated according to the presentation-description data, and the representative images of the moving-image data and the representative image of the modified data can be distinguished from each other.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to image processing apparatuses and imageprocessing methods, and more particularly, to processing ofpresentation-description data used for controlling the presentation ofimage data.

2. Description of the Related Art

Units for handling digital moving-image data, such as digital videorecorders and DVD players, are wide spread nowadays, and suchmoving-image data has been edited not only in studios but also in homesas audio-visual equipment and computer equipment have become moreadvanced.

In general moving-image editing, a part of a body of moving-image dataserving as material for editing is cut out and rearranged, and variousvideo effects are applied. As the performance of playback units has beenimproved, there is now a system in which the playback sequence ofmoving-image data is described by the use of a presentation-descriptionlanguage (script), such as a synchronized multimedia integrationlanguage (SMIL), and a playback unit performs playback according to thedescription data.

FIG. 15 shows an excerpt of presentation-description data written withSMIL 2.0. In this example, mov1.mpg to mov5.mpg data is sequentiallyplayed back.

Since script editing allows cuts to be rearranged when moving image isedited by the use of such a presentation-description data, the cost ofan editing machine and time required for editing can be largely reduced.Depending on a language specification, a video effect can be appliedduring playback. In addition, a desired effect can be applied whencontrol is applied such that moving-image data is created only for aportion where the effect is applied, during editing, and thismoving-image data is inserted into a larger body of moving-image dataserving as material for editing, by the use of presentation-descriptiondata, and played back.

Let it be assumed in FIG. 15 that mov3.mpg has moving-image data of onlya portion where a wipe effect has been applied, from a time one secondbefore the end of mov2.mpg to a time one second after the start ofmov4.mpg. When such moving-image data to which an effect has beenapplied is used, since the presentation-description data merelyindicates a usual or standard cut, editing can be performed with effect,irrespective of playback units.

There has been known an editing-screen style called a storyboard, as auser interface for editing in moving-image editing for consumers. FIG.14 shows a storyboard which illustrates mov1.mpg to mov5.mpg shown inFIG. 15. In a storyboard, thumbnail images representing moving-imageclips are displayed, and the user rearranges thumbnails which indicatemoving-image cuts, on the storyboard, to execute moving-image editing.

Even when (as in the example described above) mov3.mpg has moving-imagedata of only a portion where a wipe effect has been applied from a timeone second before the end of mov2.mpg to a time one second after thestart of mov4.mpg, since presentation-description data used in aconventional system does not have information indicating that mov3.mpgcorresponds to a portion where the wipe effect has been applied, thethumbnail image indicated by 1404 in FIG. 14 is shown on the storyboardas a usual video clip (i.e., one which does not reflect the wipe).

Therefore, the user cannot easily perceive that the clip correspondingto thumbnail 1404 is one where the wipe (or other specified) effect hasbeen applied, and editing such as to change the type of an effect, isdifficult.

SUMMARY OF THE INVENTION

It is an object of the present invention to solve the foregoing problem.

Another object of the present invention is to allow modified data to beeasily recognized when image data is processed with the use ofpresentation-description data.

One of the foregoing objects is achieved in one aspect of the presentinvention through the provision of an image processing apparatus forprocessing a plurality of items of moving-image data, modified dataobtained by modifying the plurality of items of moving-image data, andpresentation-description data which indicates the presentation order ofthe plurality of items of moving-image data and the modified data. Theapparatus comprises detection means for detecting a modified-data objectwhich specifies the presentation of the modified data, among thepresentation-description data, and representative-image generation meansfor generating usual representative-image data for moving-image data forwhich presentation is specified in the presentation-description data,and modified representative-image data for the modified data,Editing-screen generation means are provided for generatingediting-screen data formed of the usual representative-image data andthe modified representative-image data according to the output of thedetection means, as are display control means, for displaying an editingscreen based on the editing-screen data generated by the editing-screengeneration means, on display means.

One of the foregoing objects is achieved in another aspect of thepresent invention through the provision of an image processing apparatusfor processing a plurality of items of moving-image data, modified dataobtained by modifying the plurality of items of moving-image data, andpresentation-description data which indicates the presentation order ofthe plurality of items of moving-image data and the modified data. Thisapparatus comprises editing-screen generation means for generating anediting screen formed of representative images of moving-image data forwhich presentation is specified in the presentation-description data,according to the presentation-description data, and control means fordetecting a modified-data object which specifies the presentation of themodified data, among the presentation-description data, and forcontrolling the editing-screen generation means so as to disable thedisplay of a representative image of the modified data specified by themodified-data object.

One of the foregoing objects is achieved in yet another aspect of thepresent invention through the provision of an image processing apparatusfor processing a plurality of items of moving-image data, modified dataobtained by modifying the plurality of items of moving-image data, andpresentation-description data which indicates the presentation order ofthe plurality of items of moving-image data and the modified data. Thisapparatus includes editing-screen generation means, which generate anediting screen formed of representative images of moving-image data forwhich presentation is specified in the presentation-description data,and of a representative image of the modified data, according to thepresentation-description data such that the representative images of themoving-image data and the representative image of the modified data canbe distinguished from each other.

One of the foregoing objects is achieved in yet still another aspect ofthe present invention through the provision of an image processingmethod for processing a plurality of items of moving-image data,modified data obtained by modifying the plurality of items ofmoving-image data, and presentation-description data which indicates thepresentation order of the plurality of items of moving-image data andthe modified data. This method includes detecting a modified-data objectwhich specifies the presentation of the modified data, among thepresentation-description data, generating usual representative-imagedata for moving-image data for which presentation is specified in thepresentation-description data, and modified representative-image datafor the modified data, and generating editing-screen data formed of theusual representative-image data and the modified representative-imagedata according to the output of the detection step. The method alsoincludes a display control step, of displaying an editing screen basedon the editing-screen data, on display means.

One of the foregoing objects is achieved in a further aspect of thepresent invention through the provision of an image processing methodfor processing a plurality of items of moving-image data, modified dataobtained by modifying the plurality of items of moving-image data, andpresentation-description data which indicates the presentation order ofthe plurality of items of moving-image data and the modified data. Thismethod includes generating an editing screen formed of representativeimages of moving-image data for which presentation is specified in thepresentation-description data, according to the presentation-descriptiondata, and a control step of detecting a modified-data object whichspecifies the presentation of the modified data, among thepresentation-description data, and of controlling the editing-screengeneration step so as to disable the display of a representative imageof the modified data specified by the modified-data object.

One of the foregoing objects is achieved in a yet further aspect of thepresent invention through the provision of an image processing methodfor processing a plurality of items of moving-image data, modified dataobtained by modifying the plurality of items of moving-image data, andpresentation-description data which indicates the presentation order ofthe plurality of items of moving-image data and the modified data. Thismethod includes generation of an editing screen formed of representativeimages of moving-image data for which presentation is specified in thepresentation-description data, and of a representative image of themodified data, according to the presentation-description data, and therepresentative images of the moving-image data and the representativeimage of the modified data can be distinguished from each other.

Further objects, features and advantages of the present invention willbecome apparent from the following description of the preferredembodiments (with reference to the attached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an editing apparatus according to anembodiment of the present invention.

FIG. 2 is a block diagram of a display-screen generation section in theediting apparatus shown in FIG. 1.

FIG. 3 is a view of example presentation-description data.

FIG. 4 is a view showing a storyboard display screen.

FIG. 5 is a block diagram of another display-screen generation section.

FIG. 6 is a view of another example presentation-description data.

FIG. 7 is a view showing another storyboard display screen.

FIG. 8 is a block diagram of yet another display-screen generationsection.

FIG. 9 is a view showing yet another storyboard display screen.

FIG. 10 is a block diagram of still another display-screen generationsection.

FIG. 11 is a view showing still another storyboard display screen.

FIG. 12 is a block diagram of still yet another display-screengeneration section.

FIG. 13 is a view showing still yet another storyboard display screen.

FIG. 14 is a view showing a conventional storyboard display screen.

FIG. 15 is a view showing conventional example presentation-descriptiondata.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments of the present invention will be described below byreferring to the drawings.

FIG. 1 is a block diagram of an editing apparatus 100 according to anembodiment of the present invention.

In FIG. 1, the editing apparatus 100 includes an operation input section101, such as a remote controller, a mouse, or a keyboard; a controlsection 102 for controlling the operation of the editing apparatus 100according to an instruction sent from the operation input section 101; ahard disk drive (HDD) 103 for storing data handled by the editingapparatus 100, such as image data, audio data, andpresentation-description data; an input and output section 104 forinputting and outputting data to and from an external unit, the section104 including an IEEE-1394 interface or a USB interface; and anexternal-storage interface 105 for inputting and outputting data to andfrom an external storage device D, such as a disk medium, including aCD-ROM, a DVD-ROM, or a DVD-RAM, and a memory card.

The editing apparatus 100 also includes a display section 106, such as aCRT or a liquid-crystal display; a display control section 107 forcontrolling the display operation of the display section 106; a CODEC108 for encoding and decoding image data and audio data; adisplay-screen generation section 109 for generating storyboard-typedisplay screens, as described later; a modified-data processing section110 for processing presentation-description data and modified dataaccording to a user's editing operation; and a data bus 111 fortransferring data among the sections of the editing apparatus 100.

The editing apparatus 100 can use presentation-description data to editdata stored in the HDD 103. Editing in the editing apparatus 100 will bedescribed below with display-screen control being focused on.

When the operation input section 101 sends an instruction for displayinga storyboard according to presentation-description data, the controlsection 102 controls the HDD 103 so that presentation-description datastored in the HDD 103 is read and output to the display-screengeneration section 109.

FIG. 3 shows an excerpt indicating a description of a moving-imageobject related to the present invention, from presentation-descriptiondata described by the use of an extended description language based onSMIL 2.0. In FIG. 3, description elements and syntax not related to thedescription of the present embodiment are omitted. An element whichspecifies a moving image in presentation-description data is hereinaftercalled a moving-image object, and a specified moving-image file ishereinafter called moving-image data. Each line in FIG. 3 corresponds toa moving-image object, and mov1.mpg to mov5.mpg correspond tomoving-image data.

In this example, mov1.mpg to mov5.mpg are sequentially played back, andmov3.mpg is modified moving-image data of only a portion where the wipeeffect has been applied from the time one second before the end ofmov2.mpg to the time one second after the start of mov4.mpg. Themoving-image object in the third line of FIG. 3, which specifiesmov3.mpg, is called a modified-data object in the present embodiment. Asattribute information indicating that an object is a modified-dataobject, a description element of “rclipFor” is attached to the object.In this example, the attribute information “rclipFor” has a value of“transition,” which indicates a video transition effect.

FIG. 2 is a view showing the structure of circuits for generating astoryboard display screen according to presentation-description data inthe display-screen generation section 109.

In FIG. 2, a presentation-description parser circuit 202 analyzes thesyntax of presentation-description data input through a terminal 201,and extracts each moving-image object. A timetable generation circuit203 generates a timetable for presenting the moving-image data specifiedby each moving-image object according to the presentation time of eachmoving-image object analyzed by the presentation-description parsercircuit 202. The entries of the generated timetable are sequentiallysent to a data-symbol generation circuit 207, and converted to symbolimages indicating the entries in an editing screen display. Amoving-image-data thumbnail is an example of symbol image. Thedata-symbol generation circuit 207 requests a display-data acquisitioncircuit 209 to send thumbnail data corresponding to the entries, ifnecessary. The thumbnail images of moving-image clips are stored inadvance in the HDD 103 together with the moving-image data. Thedisplay-data acquisition circuit 209 outputs a control signal to the HDD103, reads the thumbnail image corresponding to a requested moving-imagedata, and sends it to the data-symbol generation circuit 207.

A modified data-object detection circuit 204 checks each moving-imageobject analyzed by the presentation-description parser circuit 202 for amodified-data identification attribute to detect a modified-data object.A switch 208 is switched for each entry of the timetable generated bythe timetable generation circuit 203 to send symbol-image data generatedby the data-symbol generation circuit 207 to a display-screen generationcircuit 210 for a usual moving-image object and to send modified-datasymbol generated by a modified-data-symbol generation circuit 206 to thedisplay-screen generation circuit 210 for a modified-data object.

The modified-data-symbol generation circuit 206 generates, for example,an image symbol indicating a video effect.

A layout control circuit 205 controls the display-screen generationcircuit 210 so as to generate a display screen known as a storyboard innon-linear video editing, on which thumbnail images are arranged in apresentation order, from the timetable output from the timetablegeneration circuit 203. In response to instructions sent from the layoutcontrol circuit 205, the display-screen generation circuit 210 arrangesthe image symbols of the entries of the timetable, sent from the switch208, generates data indicating a display screen, and outputs it to aterminal 211.

The layout control circuit 205 also arranges an image symbol indicatingan effect between usual thumbnail display positions so as to clearlyindicate the effect, according to the output of the modified-data-objectdetection circuit 204, for a modified-data object.

FIG. 4 is a view of a storyboard-type editing screen generated by theforegoing processing for the example presentation-description shown inFIG. 3.

In FIG. 4, the thumbnail images 402, 403, 405, and 406 of mov1.mpg,mov2.mpg, mov4.mpg, and mov5.mpg, respectively are shown in a storyboarddisplay window 401. These are generated by the data-symbol generationcircuit 207.

There is shown an image symbol 404 corresponding to the modified-dataobject which specifies mov3.mpg. Since the moving-image object on thethird line in FIG. 3, which specifies mov3.mpg, is a modified-dataobject to which a modified-data identification attribute has beenattached, the modified-data-object detection circuit 204 detects thismodified-data object, and controls the switch 208 so as to output theimage symbol indicating the modified-data object to the display-screengeneration circuit 210. In response to the output of themodified-data-object detection circuit 204, the layout control circuit205 places the image symbol 404 between the thumbnail images 403 and405, which corresponds to usual moving-image objects, so as to clearlyshows that the modified-data object indicates an effect, at thepresentation timing of the modified-data object corresponding to theimage symbol 404.

In this way, in the present embodiment, since, for modified data whichexpresses only a video effect, a symbol icon indicating the videoeffect, not a usual moving-image-data display, is displayed in anediting operation screen which is like a storyboard, the user can easilyunderstand the position of the video-effect image.

The user checks the storyboard based on the presentation-descriptiondata. When the user wants to change the presentation order, the userspecifies an editing mode on the operation input section 101. When aninstruction for the editing mode is issued, the control section 102activates an editing application. The display-screen generation section109 generates an editing screen, and the screen is displayed on thedisplay section 106. The user performs operations on the editing screenby the use of the operation input section 101 to apply edit processingto moving-image data and audio data stored in the HDD 103. The editingprocessing can be performed with a known structure. The modified-dataprocessing section 110 generates modified data obtained by applyingeffect processing to image data.

When the editing processing is finished, the control section 102controls the modified-data processing section 110 to generatepresentation-description data according to the presentation orderobtained by editing. When modified data is disposed in the presentationorder, attribute information is added to the modified-data object in thepresentation-description data, as shown in FIG. 3. The generatedpresentation-description data is stored in the HDD 103. The user canspecify the storyboard display of the presentation-description data todisplay a storyboard screen based on the generatedpresentation-description data, on the display section 106 and to checkthe result of editing.

A second embodiment will be described next, in which a storyboard screenbased on presentation-description data is shown differently. In thefollowing embodiment, the same editing apparatus as that shown in FIG. 1is used.

FIG. 5 is a view showing a main section of a display-screen generationsection 109. The same numbers as those used in FIG. 2 are assigned tothe same portions as those shown in FIG. 2.

In the same way as described above, presentation-description data to beedited is input to a terminal 201.

FIG. 6 shows an excerpt indicating a description of a moving-imageobject related to the present embodiment, from presentation-descriptiondata described by the use of an extended description language based onSMIL 2.0. In the example shown in FIG. 6, mov1.mpg to mov5.mpg aresequentially played back, and mov3.mpg is modified moving-image data ofonly a portion where the wipe effect has been applied from the time onesecond before the end of mov2.mpg to the time one second after the startof mov4.mpg.

In the present embodiment, a description element of “rclipFor” isattached to an object on the third line in FIG. 6 as attributeinformation indicating that the object is a modified-data object. Thevalue of the attribute information “rclipFor” is set to a modificationtype of “wipe_LR,” which indicates a wipe effect from the left to theright on the screen. In the present embodiment, this video effect typedetermines a modified-data image symbol to be displayed on a storyboard.

In FIG. 5, a presentation-description parser circuit 202 analyzes thesyntax of the input presentation-description data, and extracts eachmoving-image object. A timetable generation circuit 203 generates atimetable, as described earlier, and sequentially sends the entries ofthe generated timetable to a data-symbol generation circuit 207. Thedata-symbol generation circuit 207 generates thumbnail moving-image dataas a symbol image expressing each entry on the editing screen.

A modified-data-object detection circuit 204 detects a modified-dataobject from each moving-image object analyzed by thepresentation-description parser circuit 202, and switches a switch 208for each entry of the timetable to send a symbol image generated by thedata-symbol generation circuit 207 and a modified-data symbol imagegenerated by a modified-data-symbol generation circuit 206 to adisplay-screen generation circuit 210.

A modification-type determination circuit 501 checks a modified-dataobject detected by the modified-data-object detection circuit 204 forthe value of the modified-data-object identification attribute todetermine its modification type, and sends a signal indicating themodification type to a modified-data-symbol generation circuit 206. Themodified-data-symbol generation circuit 206 generates a symbol iconcorresponding to the determined modification type, such as a symbol iconfor the wipe effect for the case shown in FIG. 6.

A layout control circuit 205 controls the display-screen generationcircuit 210 so as to generate a storyboard screen from the timetableoutput from the timetable generation circuit 203. In response toinstructions sent from the layout control circuit 205, thedisplay-screen generation circuit 210 arranges the image symbols of theentries of the timetable, sent from the switch 208, to generateediting-screen data, and outputs it to a terminal 211. An image symbolfor a modified-data object is placed between usual thumbnail images soas to clearly indicate that the object corresponds to an effect.

FIG. 7 is a view of a storyboard-type editing screen generated by theforegoing processing for the example presentation-description shown inFIG. 6. The thumbnail images 702, 703, 705, and 706 of mov1.mpg,mov2.mpg, mov4.mpg, and mov5.mpg, respectively, are shown in astoryboard display window 701. These are generated by the data-symbolgeneration circuit 207.

The moving-image object specifying mov3.mpg on the third line in FIG. 6is a modified-data object to which a modified-data identificationattribute has been attached. For this modified-data object, amodified-data symbol 704 indicating a wipe effect from the left to theright on the screen, generated by the modified-data-symbol generationcircuit 206, is displayed. In response to the output of themodified-data-object detection circuit 204, the layout control circuit205 places the modified-data symbol image 704 between the usualthumbnail images 703 and 705 so as to clearly shows that themodified-data object indicates an effect.

In this way, in the present embodiment, for moving-image data whichexpresses only a video effect, a symbol icon with which even an effectcontent can be identified, not a usual moving-image-data display, isdisplayed in an editing operation screen which is like a storyboard.

Since an effect content is determined from the attribute information ofa modified-data object, and a symbol image indicating the effect contentis generated and displayed, the user can easily confirm the content ofthe modification applied to the modified data.

A third embodiment will be described next, in which a storyboard screenbased on presentation-description data is shown further differently. Inthe following embodiment, the same editing apparatus as that shown inFIG. 1 is used.

FIG. 8 is a view showing a main section of a display-screen generationsection 109. The same numbers as those used in FIG. 2 and FIG. 5 areassigned to the same portions as those shown in FIG. 2 and FIG. 5.

In FIG. 8, presentation-description data to be edited is input to aterminal 201. A presentation-description parser circuit 202 analyzes thesyntax of the input presentation-description data, such as that shown inFIG. 3 or FIG. 6, and extracts each moving-image object. A timetablegeneration circuit 203 generates a timetable for presenting themoving-image data specified by each moving-image object according to thepresentation time of each moving-image object analyzed by thepresentation-description parser circuit 202.

The entries of the generated timetable are sequentially sent through aswitch 801 to a modified data-symbol generation circuit 206 or to adata-symbol generation circuit 207, and a symbol image expressing eachentry on the editing screen is generated by the modified-data-symbolgeneration circuit 206 or by the data-symbol generation circuit 207.

A modified-data-object detection circuit 204 checks each moving-imageobject analyzed by the presentation-description parser circuit 202 for amodified-data identification attribute to detect a modified-data object.The modified-data-object detection circuit 204 controls the switch 801such that, when an entry in the timetable is a modified-data object, theoutput of the timetable generation circuit 203 is sent to themodified-data-symbol generation circuit 206, and for other cases, theoutput of the timetable generation circuit 203 is sent to thedata-symbol generation circuit 207.

The data-symbol generation circuit 207 generates thumbnail imagesexpressing entries on the editing screen for usual data objects in thetimetable.

The modified-data-symbol generation circuit 206 generates modified-datasymbol images expressing entries on the editing screen for modified-dataobjects in the timetable, and sends them to the switch 208. In thepresent embodiment, a symbol icon indicating modified data is superposedon a thumbnail image for the modified data to generate a symbol imagefor the modified data.

To this end, the modified-data-symbol generation circuit 206 requests adisplay-data acquisition circuit 209 to send the thumbnail image datacorresponding to an entry, if necessary. The display-data acquisitioncircuit 209 reads the thumbnail image data corresponding to therequested moving-image data from the HDD 103, and sends it to themodified-data-symbol generation circuit 206. The modified-data-symbolgeneration circuit 206 superposes a symbol icon indicating a videoeffect on the thumbnail image data to generate a modified-data symbolimage.

A layout control circuit 205 controls the display-screen generationcircuit 210 so as to generate an editing screen on which thumbnailimages are arranged in a presentation order, from the timetable. Inresponse to instructions sent from the layout control circuit 205, thedisplay-screen generation circuit 210 arranges the symbol images of theentries of the timetable, sent from the switch 208, generatesediting-screen data, and outputs it to a terminal 211.

FIG. 9 is a view of a storyboard-type editing screen generated by theforegoing processing for the example presentation-description shown inFIG. 3. In FIG. 9, the thumbnail images 902, 903, 905, and 906 ofmov1.mpg, mov2.mpg, mov4.mpg, and mov5.mpg, respectively, are shown in astoryboard display window 901. These are generated by the data-symbolgeneration circuit 207. For the modified-data object, a thumbnail image904 in which a modified-data symbol 907 indicating a video effect,generated by the modified-data-symbol generation circuit 206, has beensuperposed is displayed.

In this way, in the present embodiment, since, for modified data whichexpresses only a video effect, a thumbnail image which includes a symbolicon indicating the video effect, not a usual moving-image-data display,is displayed in an editing operation screen which is like a storyboard,the user can easily recognize the modified data specified by thepresentation-description data.

A fourth embodiment will be described next, in which a storyboard screenbased on presentation-description data is shown still furtherdifferently. Also in the following embodiment, the same editingapparatus as that shown in FIG. 1 is used.

FIG. 10 is a view showing a main section of a display-screen generationsection 109. The same numbers as those used in FIG. 2, FIG. 5, and FIG.8 are assigned to the same portions as those shown in FIG. 2, FIG. 5,and FIG. 8.

In FIG. 10, a presentation-description data to be edited is input to aterminal 201. A presentation-description parser circuit 202 analyzes thesyntax of the input presentation-description data, and extracts eachmoving-image object. A timetable generation circuit 203 generates atimetable for presenting the moving-image data specified by eachmoving-image object according to the presentation time of eachmoving-image object analyzed by the presentation-description parsercircuit 202.

The entries of the generated timetable are sequentially sent through aswitch 801 to a modified-data-symbol generation circuit 206 or to adata-symbol generation circuit 207, and a symbol image expressing eachentry on the editing screen is generated by the modified-data-symbolgeneration circuit 206 or by the data-symbol generation circuit 207.

A modified-data-object detection circuit 204 checks each moving-imageobject for a modified-data identification attribute to detect amodified-data object. The modified-data-object detection circuit 204controls the switch 801 such that, when an entry in the timetable is amodified-data object, the output of the timetable generation circuit 203is sent to the modified-data-symbol generation circuit 206, and forother cases, the output of the timetable generation circuit 203 is sentto the data-symbol generation circuit 207.

The data-symbol generation circuit 207 generates thumbnail imagesexpressing entries on the editing screen for usual data objects in thetimetable, and sends them to a display-screen generation circuit 210through the switch 208.

The modified-data-symbol generation circuit 206 generates modified-datasymbol images expressing entries on the editing screen for modified-dataobjects in the timetable, and sends them to the switch 208. In thepresent embodiment, a symbol icon indicating a video-effect type issuperposed on a thumbnail image for moving-image data to generate asymbol image for a modified-data object.

To this end, the modified-data-symbol generation circuit 206 requests adisplay-data acquisition circuit 209 to send the thumbnail image datacorresponding to an entry, if necessary. The display-data acquisitioncircuit 209 reads the thumbnail image data corresponding to therequested moving-image data from the HDD 103, and sends it to themodified-data-symbol generation circuit 206.

A modification-type determination circuit 501 checks a detectedmodified-image object for the value of a modified-data-objectidentification attribute to determine the type of the modification, andsends it to the modified-data-symbol generation circuit 206. Themodified-data-symbol generation circuit 206 determines a modified-datasymbol according to the determined video-effect type. In thepresentation-description data shown in FIG. 6, for example, since themodification type of the modified-data object is “wipe,” a symbol iconfor the wipe effect is superposed on the thumbnail image data togenerate a modified-data symbol.

A layout control circuit 205 controls the display-screen generationcircuit 210 so as to generate an editing screen on which the thumbnailimages of the entries are arranged in a presentation order, from thetimetable. Under the control, the display-screen generation circuit 210arranges the symbol images of the entries of the timetable, sent fromthe switch 208, to generate editing-screen data, and outputs it to aterminal 211.

FIG. 11 is a view of a storyboard-type editing screen generated by theforegoing processing for the example presentation-description shown inFIG. 6.

The thumbnail images 1102, 1103, 1105, and 1106 of mov1.mpg, mov2.mpg,mov4.mpg, and mov5.mpg, respectively, are shown in a storyboard displaywindow 1101. For a modified-data object, a thumbnail image 1104 in whicha modified-data symbol 1107 indicating a wipe effect from the left tothe right on the screen, generated by the modified-data-symbolgeneration circuit 206, has been superposed is displayed.

In this way, in the present embodiment, for modified data whichexpresses only a video effect, a thumbnail image which includes a symbolicon with which even an effect content can be identified, not a usualmoving-image-data display, is displayed in an editing operation screenwhich is like a storyboard. Therefore, the user can easily recognizemodified data specified by presentation-description data, and thecontent of the modification applied to the modified data.

A fifth embodiment will be described next, in which a storyboard screenbased on presentation-description data is shown yet further differently.Also in the following embodiment, the same editing apparatus as thatshown in FIG. 1 is used.

FIG. 12 is a view showing a main section of a display-screen generationsection 109. The same numbers as those used in FIG. 2 are assigned tothe same portions as those shown in FIG. 2.

In FIG. 12, a presentation-description data to be edited is input to aterminal 201. A presentation-description parser circuit 202 analyzes thesyntax of the input presentation-description data, and extracts eachmoving-image object. A timetable generation circuit 203 generates atimetable for presenting the moving-image data specified by eachmoving-image object according to the presentation time of eachmoving-image object analyzed by the presentation-description parsercircuit 202. The entries of the generated timetable are sequentiallysent to a data-symbol generation circuit 207.

The data-symbol generation circuit 207 generates thumbnail imagesexpressing the entries of the timetable, and sends them to adisplay-screen generation circuit 210 through a switch 208.

A modified-data-object detection circuit 204 checks each moving-imageobject for a modified-data identification attribute to detect amodified-data object. On/off control is applied to the switch 208 foreach entry of the timetable such that, for a usual moving-image object,a symbol image generated by the data-symbol generation circuit 207 issent to the display-screen generation circuit 210, and for amodified-data object, a symbol image is not sent.

A layout control circuit 205 controls the display-screen generationcircuit 210 so as to generate an editing screen on which thumbnailimages are arranged in a presentation order, from the timetable. Inresponse to instructions sent from the layout control circuit 205, thedisplay-screen generation circuit 210 arranges the symbol images(thumbnail images) of the entries of the timetable, sent from the switch208, generates editing-screen data, and outputs it to a terminal 211.

In the present embodiment, the layout control circuit 205 controls suchthat a thumbnail image is not displayed for a modified-data object and aspace is eliminated.

FIG. 13 is a view of a storyboard-type editing screen generated by theforegoing processing for the example presentation description shown inFIG. 3. In FIG. 13, the thumbnail images 1302, 1303, 1305, and 1306 ofmov1.mpg, mov2.mpg, mov4.mpg, and mov5.mpg, respectively, are shown in astoryboard display window 1301. The thumbnail image of a modified-dataobject is not displayed because the modified-data-object detectioncircuit 204 turns off the switch 208 so as to disable the output of thethumbnail image of the modified-data object, and the layout controlcircuit 205 controls such that a space between the thumbnail images 1303and 1305 shown in FIG. 13 is removed.

According to the present embodiment, moving-image data expressing only avideo effect is not displayed in the storyboard. Therefore, the user isprevented from erroneously taking a modified-data object for a usualmoving-image object.

In the foregoing embodiments, SMIL is used as a presentation-descriptionlanguage. A presentation order may be controlled with other descriptionlanguages, such as an extensible markup language (XML).

In the foregoing embodiments, image data and presentation-descriptiondata are stored in an HDD and processed. The present invention can alsobe applied to a case in which presentation-description data is stored ina medium corresponding to the external storage device shown in FIG. 1,such as an optical disk or a memory card, together with moving-imagedata and audio data, and processed.

In the foregoing embodiments, the wipe processing in thepresentation-description data shown in FIG. 6 is taken as an example. Itis also possible to apply other effect processing. In addition, amodification such as title synthesis can also be applied.

In the foregoing embodiments, the present invention is applied to theapparatus shown in FIG. 1. It is also possible that the same functionsas those of the apparatus are implemented by a microcomputer and itsperipheral units. In this case, the present invention includes equipmentformed of the microcomputer and its peripheral units, and a memory whichstores software for controlling the microcomputer.

As described above, according to the present invention, modified datacan be easily recognized in a case in which information data isprocessed with the use of presentation-description data.

While the present invention has been described with reference to whatare presently considered to be the preferred embodiments, it is to beunderstood that the invention is not limited to the disclosedembodiments. On the contrary, the invention is intended to cover variousmodifications and equivalent arrangements included within the spirit andscope of the appended claims. The scope of the following claims is to beaccorded the broadest interpretation so as to encompass all suchmodifications and equivalent structures and functions.

1. An image processing apparatus for displaying a storyboard that laysout along one dimension a plurality of thumbnail images each of whichrepresents a video clip, and for processing presentation-descriptiondata which designates a presentation order of a plurality of video clipsincluding a video clip of an item of moving-image data and of a videoclip of modified image data, different from the moving image data,obtained by modifying any of a plurality of items of the moving imagedata, comprising: a symbol image generation unit configured to generatesymbol image data different from thumbnail image data representing thevideo clip of the modified image data, for the video clip of themodified image data; a detection unit configured to detect the videoclip of the modified image data from the plurality of video clipsdesignated by the presentation-description data according to thepresentation-description data; an editing-screen generation unitconfigured to generate editing-screen data for an editing screendisplaying the storyboard that lays out the plurality of thumbnailimages of the video clips in an order according to the presentationorder designated by the presentation-description data, theediting-screen generation unit laying out, if the video clip of themodified image data is detected by the detection unit, a symbol imagerepresented by the symbol image data of the video clip of the modifiedimage data generated by the symbol image generation unit instead of thethumbnail image of the video clip of the modified image data in thestoryboard; and a display control unit configured to display the editingscreen based on the editing-screen data generated by said editing-screengeneration unit, on a display device.
 2. An image processing apparatusaccording to claim 1, wherein said symbol image generation unit detectsa modification type assigned to the modified image data according toattribute information included in the presentation-description data, andgenerates the symbol image data for the video clip of the modified dataindicating the detected modification type.
 3. An image processingapparatus according to claim 2, wherein said symbol image generationunit generates a symbol icon indicating the modification type, whereinthe modification type comprises a video effect.
 4. An image processingapparatus according to claim 3, wherein said symbol image generationunit superimposes the symbol icon on a thumbnail image of the video clipof the modified image data, generating superposition image data, andoutputs the generated superposition image data as the symbol image forthe modified image data.
 5. An image processing apparatus according toclaim 1, wherein said symbol image generation unit generates a symbolicon indicating the modified image data, and outputs the symbol icon asthe symbol image for the modified image data.
 6. An image processingapparatus according to claim 1, wherein said symbol image generationunit superimposes an icon indicating the modified image data on athumbnail image representing the modified image data, generatingsuperposition image data, and outputs the generated superposition imagedata as the symbol image for the modified image data.
 7. An imageprocessing apparatus according to claim 1, wherein said editing-screengeneration unit arranges the thumbnail images at positions correspondingto their presentation order.
 8. An image processing apparatus accordingto claim 7, wherein the editing-screen generation unit detects the videoclips of items of the moving-image data which are presented before andafter the video clip of the modified image data according to thepresentation-description data and places the symbol image of the videoclip of the modified image data between the thumbnail image of the videoclips of the moving-image data presented before and after the video clipof the modified image data in the storyboard.
 9. An image processingapparatus according to claim 1, wherein the editing-screen generationunit includes an acquisition unit configured to acquire thumbnail imagedata of the video clips and generates the editing-screen data by usingthe thumbnail image data acquired by the acquisition unit.
 10. An imageprocessing apparatus according to claim 9, wherein the acquisition unitreads the thumbnail image data from a storage medium in which themoving-image data has been stored.
 11. An image processing apparatusaccording to claim 1, wherein the moving-image data and the modifiedimage data are recorded in a single recording medium.
 12. An imageprocessing apparatus according to claim 11, wherein thepresentation-description data is recorded in the recording medium. 13.An image processing apparatus comprising: a display apparatus thatdisplays a storyboard that lays out along one dimension a plurality ofthumbnail images each of which represents a video clip; an apparatusthat processes presentation-description data which designates thepresentation order of a plurality of video clips including a video clipof an item of moving-image data and a video clip of modified image data,different from the moving-image data, obtained by modifying any of aplurality of items of the moving-image data, wherein said displayapparatus displays the storyboard that lays out the plurality ofthumbnail images of the video clips in an order according to thepresentation order designated by the presentation-description data, andwherein when said image processing apparatus detects the video clip ofthe modified image data from the plurality of video clips designated bythe presentation-description data, said display apparatus lays out asymbol image different from the thumbnail image representing the videoclip of the modified image data instead of the thumbnail image of thevideo clip of the modified image data in the storyboard.
 14. An imageprocessing method for displaying a storyboard that lays out along onedimension a plurality of thumbnail images each of which represents avideo clip, and for processing presentation-description data whichdesignates a presentation order of a plurality of video clips includinga video clip of an item of moving-image data and of a video clip ofmodified image data, different from the moving-image data, obtained bymodifying any of a plurality of items of the moving-image data,comprising: a symbol image generation step for generating symbol imagedata different from thumbnail image data representing the video clip ofthe modified image data, for the video clip of the modified image data;a detection step of detecting the video clip of the modified image datafrom the plurality of video clips designated by thepresentation-description data according to the presentation-descriptiondata; an editing-screen generation step, of generating editing-screendata for an editing screen displaying the storyboard that lays out theplurality of thumbnail images of the video clips in an order accordingto the presentation order designated by the presentation-descriptiondata, the editing-screen generation step laying out, if the video clipof the modified image data is detected by the detection step, a symbolimage represented by the symbol image data of the video clip of themodified image data generated by the symbol image generation stepinstead of the thumbnail image of the video clip of the modified imagedata in the storyboard; and a display control step, of displaying theediting screen based on the editing-screen data, on a display device.15. An image processing method for an image processing apparatus,comprising: displaying a storyboard that lays out along one dimension aplurality of thumbnail images each of which represents a video clip; andprocessing presentation-description data which designates thepresentation order of a plurality of video clips including a video clipof an item of moving-image data and of a video clip of modified imagedata, different from the moving-image data, obtained by modifying any ofa plurality of items of the moving-image data, wherein said displayingstep displays the storyboard that lays out the plurality of thumbnailimages of the video clips in an order according to the presentationorder designated by the presentation-description data, and wherein whenthe image processing apparatus detects the video clip of the modifiedimage data from the plurality of video clips designated bypresentation-description data, the displaying step lays out a symbolimage different from the thumbnail image representing the video clip ofthe modified image data instead of the thumbnail image of the video clipof the modified image data in the storyboard.